<html>
<head><meta charset="utf-8"><title>slowdown compiling Cargo · t-compiler/wg-parallel-rustc · Zulip Chat Archive</title></head>
<h2>Stream: <a href="https://rust-lang.github.io/zulip_archive/stream/187679-t-compiler/wg-parallel-rustc/index.html">t-compiler/wg-parallel-rustc</a></h2>
<h3>Topic: <a href="https://rust-lang.github.io/zulip_archive/stream/187679-t-compiler/wg-parallel-rustc/topic/slowdown.20compiling.20Cargo.html">slowdown compiling Cargo</a></h3>

<hr>

<base href="https://rust-lang.zulipchat.com">

<head><link href="https://rust-lang.github.io/zulip_archive/style.css" rel="stylesheet"></head>

<a name="176505659"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187679-t-compiler/wg-parallel-rustc/topic/slowdown%20compiling%20Cargo/near/176505659" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Alex Crichton <a href="https://rust-lang.github.io/zulip_archive/stream/187679-t-compiler/wg-parallel-rustc/topic/slowdown.20compiling.20Cargo.html#176505659">(Sep 24 2019 at 20:59)</a>:</h4>
<p>I was wondering, is this an alright place to discuss perf impact of parallel rustc nowadays? Or is an issue a better place? I just built a parallel rustc locally and compared it with the previous nightly, and I was curious to test out <code>-Z timings</code> and see what cpu usage looked like during the build, but it was unfortunately slower in that a parallel rustc took 74s for a full release build (nothing cached) compared to 62 on nightly</p>



<a name="176506259"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187679-t-compiler/wg-parallel-rustc/topic/slowdown%20compiling%20Cargo/near/176506259" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/187679-t-compiler/wg-parallel-rustc/topic/slowdown.20compiling.20Cargo.html#176506259">(Sep 24 2019 at 21:06)</a>:</h4>
<p>This seems like a perfect place.</p>
<p>That sounds somewhat expected, though -- I suspect we're pretty poorly tuned and overeager at grabbing jobserver tokens</p>



<a name="176506278"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187679-t-compiler/wg-parallel-rustc/topic/slowdown%20compiling%20Cargo/near/176506278" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/187679-t-compiler/wg-parallel-rustc/topic/slowdown.20compiling.20Cargo.html#176506278">(Sep 24 2019 at 21:06)</a>:</h4>
<p>You might see more success with e.g. RUSTFLAGS="-Zthreads=2"</p>



<a name="176506378"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187679-t-compiler/wg-parallel-rustc/topic/slowdown%20compiling%20Cargo/near/176506378" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Alex Crichton <a href="https://rust-lang.github.io/zulip_archive/stream/187679-t-compiler/wg-parallel-rustc/topic/slowdown.20compiling.20Cargo.html#176506378">(Sep 24 2019 at 21:07)</a>:</h4>
<p>Ok cool, so yeah taking a look at things I'm watching cpu usage on nightly and it's "all green" which I think means all in userspace</p>



<a name="176506419"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187679-t-compiler/wg-parallel-rustc/topic/slowdown%20compiling%20Cargo/near/176506419" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Alex Crichton <a href="https://rust-lang.github.io/zulip_archive/stream/187679-t-compiler/wg-parallel-rustc/topic/slowdown.20compiling.20Cargo.html#176506419">(Sep 24 2019 at 21:08)</a>:</h4>
<p>w/ parallel rustc it's "mostly red" which I think means most of the time is spent in the kernel</p>



<a name="176506442"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187679-t-compiler/wg-parallel-rustc/topic/slowdown%20compiling%20Cargo/near/176506442" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Alex Crichton <a href="https://rust-lang.github.io/zulip_archive/stream/187679-t-compiler/wg-parallel-rustc/topic/slowdown.20compiling.20Cargo.html#176506442">(Sep 24 2019 at 21:08)</a>:</h4>
<p>a quick <code>perf</code> shows a huge amount of time in the kernel</p>



<a name="176506453"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187679-t-compiler/wg-parallel-rustc/topic/slowdown%20compiling%20Cargo/near/176506453" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Alex Crichton <a href="https://rust-lang.github.io/zulip_archive/stream/187679-t-compiler/wg-parallel-rustc/topic/slowdown.20compiling.20Cargo.html#176506453">(Sep 24 2019 at 21:08)</a>:</h4>
<p>trying to track down what's where</p>



<a name="176506476"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187679-t-compiler/wg-parallel-rustc/topic/slowdown%20compiling%20Cargo/near/176506476" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Alex Crichton <a href="https://rust-lang.github.io/zulip_archive/stream/187679-t-compiler/wg-parallel-rustc/topic/slowdown.20compiling.20Cargo.html#176506476">(Sep 24 2019 at 21:08)</a>:</h4>
<p>I think the jobserver management may also be wrong?</p>



<a name="176506498"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187679-t-compiler/wg-parallel-rustc/topic/slowdown%20compiling%20Cargo/near/176506498" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Alex Crichton <a href="https://rust-lang.github.io/zulip_archive/stream/187679-t-compiler/wg-parallel-rustc/topic/slowdown.20compiling.20Cargo.html#176506498">(Sep 24 2019 at 21:08)</a>:</h4>
<p>when I ran a cargo build I got "45 (jobs=28 ncpu=28)"</p>



<a name="176506522"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187679-t-compiler/wg-parallel-rustc/topic/slowdown%20compiling%20Cargo/near/176506522" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Alex Crichton <a href="https://rust-lang.github.io/zulip_archive/stream/187679-t-compiler/wg-parallel-rustc/topic/slowdown.20compiling.20Cargo.html#176506522">(Sep 24 2019 at 21:09)</a>:</h4>
<p>er, that means that there were at most 45 rustc instances running in parallel</p>



<a name="176506538"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187679-t-compiler/wg-parallel-rustc/topic/slowdown%20compiling%20Cargo/near/176506538" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Alex Crichton <a href="https://rust-lang.github.io/zulip_archive/stream/187679-t-compiler/wg-parallel-rustc/topic/slowdown.20compiling.20Cargo.html#176506538">(Sep 24 2019 at 21:09)</a>:</h4>
<p>but the default, <code>-j28</code>, should have made it such that no more than 28 rustc instances were running</p>



<a name="176506561"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187679-t-compiler/wg-parallel-rustc/topic/slowdown%20compiling%20Cargo/near/176506561" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Alex Crichton <a href="https://rust-lang.github.io/zulip_archive/stream/187679-t-compiler/wg-parallel-rustc/topic/slowdown.20compiling.20Cargo.html#176506561">(Sep 24 2019 at 21:09)</a>:</h4>
<p>nightly does indeed not exceed 28</p>



<a name="176506614"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187679-t-compiler/wg-parallel-rustc/topic/slowdown%20compiling%20Cargo/near/176506614" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Alex Crichton <a href="https://rust-lang.github.io/zulip_archive/stream/187679-t-compiler/wg-parallel-rustc/topic/slowdown.20compiling.20Cargo.html#176506614">(Sep 24 2019 at 21:10)</a>:</h4>
<p>yeah a huge amount of time is spent acquiring/releaseing tokens</p>



<a name="176506663"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187679-t-compiler/wg-parallel-rustc/topic/slowdown%20compiling%20Cargo/near/176506663" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Alex Crichton <a href="https://rust-lang.github.io/zulip_archive/stream/187679-t-compiler/wg-parallel-rustc/topic/slowdown.20compiling.20Cargo.html#176506663">(Sep 24 2019 at 21:10)</a>:</h4>
<p><code>- 21.09% rustc_rayon_core::sleep::Sleep::sleep</code> from perf</p>



<a name="176506751"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187679-t-compiler/wg-parallel-rustc/topic/slowdown%20compiling%20Cargo/near/176506751" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/187679-t-compiler/wg-parallel-rustc/topic/slowdown.20compiling.20Cargo.html#176506751">(Sep 24 2019 at 21:11)</a>:</h4>
<p>I am not really surprised -- I forget what our current management strategy is, but it might be something like "let's release/reacquire tokens whenever a rayon thread goes idle, which presumably happens quite often</p>



<a name="176507750"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187679-t-compiler/wg-parallel-rustc/topic/slowdown%20compiling%20Cargo/near/176507750" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Alex Crichton <a href="https://rust-lang.github.io/zulip_archive/stream/187679-t-compiler/wg-parallel-rustc/topic/slowdown.20compiling.20Cargo.html#176507750">(Sep 24 2019 at 21:23)</a>:</h4>
<p>is this something I should open an issue for?</p>



<a name="176508100"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187679-t-compiler/wg-parallel-rustc/topic/slowdown%20compiling%20Cargo/near/176508100" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/187679-t-compiler/wg-parallel-rustc/topic/slowdown.20compiling.20Cargo.html#176508100">(Sep 24 2019 at 21:27)</a>:</h4>
<p>I think at this point I would say no</p>



<a name="176508154"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187679-t-compiler/wg-parallel-rustc/topic/slowdown%20compiling%20Cargo/near/176508154" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/187679-t-compiler/wg-parallel-rustc/topic/slowdown.20compiling.20Cargo.html#176508154">(Sep 24 2019 at 21:28)</a>:</h4>
<p>we're not really at the point where performance is a (sufficient) concern</p>



<a name="176508179"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187679-t-compiler/wg-parallel-rustc/topic/slowdown%20compiling%20Cargo/near/176508179" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/187679-t-compiler/wg-parallel-rustc/topic/slowdown.20compiling.20Cargo.html#176508179">(Sep 24 2019 at 21:28)</a>:</h4>
<p>the current jobserver management strategy is unknown (I just pulled one out of thin air)</p>



<a name="176508199"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187679-t-compiler/wg-parallel-rustc/topic/slowdown%20compiling%20Cargo/near/176508199" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/187679-t-compiler/wg-parallel-rustc/topic/slowdown.20compiling.20Cargo.html#176508199">(Sep 24 2019 at 21:28)</a>:</h4>
<p>I think Zoxc might know it but not even sure about that</p>



<a name="176508237"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187679-t-compiler/wg-parallel-rustc/topic/slowdown%20compiling%20Cargo/near/176508237" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/187679-t-compiler/wg-parallel-rustc/topic/slowdown.20compiling.20Cargo.html#176508237">(Sep 24 2019 at 21:29)</a>:</h4>
<p><span class="user-mention" data-user-id="116015">@Alex Crichton</span> It might be good to open an issue about the jobserver management strategy though -- I don't know how we should do that, and discussing on an issue seems good</p>



<a name="176508267"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187679-t-compiler/wg-parallel-rustc/topic/slowdown%20compiling%20Cargo/near/176508267" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/187679-t-compiler/wg-parallel-rustc/topic/slowdown.20compiling.20Cargo.html#176508267">(Sep 24 2019 at 21:29)</a>:</h4>
<p>in particular I think we're going to need some more intelligent server than the current model allows to evenly distribute tokens between rustc instances</p>



<a name="176508285"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187679-t-compiler/wg-parallel-rustc/topic/slowdown%20compiling%20Cargo/near/176508285" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Alex Crichton <a href="https://rust-lang.github.io/zulip_archive/stream/187679-t-compiler/wg-parallel-rustc/topic/slowdown.20compiling.20Cargo.html#176508285">(Sep 24 2019 at 21:29)</a>:</h4>
<p>ok, I'll open an issue</p>



<a name="176508348"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187679-t-compiler/wg-parallel-rustc/topic/slowdown%20compiling%20Cargo/near/176508348" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/187679-t-compiler/wg-parallel-rustc/topic/slowdown.20compiling.20Cargo.html#176508348">(Sep 24 2019 at 21:30)</a>:</h4>
<p>e.g. if we have something like 8 cores we probably want 4 rustcs each with 2 threads (approximately) rather than 1 rustc with 8 threads, I'd imagine</p>



<a name="176508648"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187679-t-compiler/wg-parallel-rustc/topic/slowdown%20compiling%20Cargo/near/176508648" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Alex Crichton <a href="https://rust-lang.github.io/zulip_archive/stream/187679-t-compiler/wg-parallel-rustc/topic/slowdown.20compiling.20Cargo.html#176508648">(Sep 24 2019 at 21:34)</a>:</h4>
<p><a href="https://github.com/rust-lang/rust/issues/64750" target="_blank" title="https://github.com/rust-lang/rust/issues/64750">https://github.com/rust-lang/rust/issues/64750</a></p>



<a name="176508993"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187679-t-compiler/wg-parallel-rustc/topic/slowdown%20compiling%20Cargo/near/176508993" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Alex Crichton <a href="https://rust-lang.github.io/zulip_archive/stream/187679-t-compiler/wg-parallel-rustc/topic/slowdown.20compiling.20Cargo.html#176508993">(Sep 24 2019 at 21:38)</a>:</h4>
<p>so additionally after focusing the call grpah a bit more</p>



<a name="176509002"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187679-t-compiler/wg-parallel-rustc/topic/slowdown%20compiling%20Cargo/near/176509002" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Alex Crichton <a href="https://rust-lang.github.io/zulip_archive/stream/187679-t-compiler/wg-parallel-rustc/topic/slowdown.20compiling.20Cargo.html#176509002">(Sep 24 2019 at 21:38)</a>:</h4>
<p>80% of rustc's time is spent in <code>__GI_clone</code></p>



<a name="176509008"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187679-t-compiler/wg-parallel-rustc/topic/slowdown%20compiling%20Cargo/near/176509008" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Alex Crichton <a href="https://rust-lang.github.io/zulip_archive/stream/187679-t-compiler/wg-parallel-rustc/topic/slowdown.20compiling.20Cargo.html#176509008">(Sep 24 2019 at 21:38)</a>:</h4>
<p>which I think is spawning threads</p>



<a name="176509021"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187679-t-compiler/wg-parallel-rustc/topic/slowdown%20compiling%20Cargo/near/176509021" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Alex Crichton <a href="https://rust-lang.github.io/zulip_archive/stream/187679-t-compiler/wg-parallel-rustc/topic/slowdown.20compiling.20Cargo.html#176509021">(Sep 24 2019 at 21:38)</a>:</h4>
<p>does rustc spawn threads on demand or immediately?</p>



<a name="176509042"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187679-t-compiler/wg-parallel-rustc/topic/slowdown%20compiling%20Cargo/near/176509042" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/187679-t-compiler/wg-parallel-rustc/topic/slowdown.20compiling.20Cargo.html#176509042">(Sep 24 2019 at 21:39)</a>:</h4>
<p>hm, I thought it was immediately, but I think we leave it up to rayon -- maybe rayon is spinning up threads if they're idle?</p>



<a name="176509050"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187679-t-compiler/wg-parallel-rustc/topic/slowdown%20compiling%20Cargo/near/176509050" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/187679-t-compiler/wg-parallel-rustc/topic/slowdown.20compiling.20Cargo.html#176509050">(Sep 24 2019 at 21:39)</a>:</h4>
<p><span class="user-mention" data-user-id="138448">@cuviper</span> might know</p>



<a name="176509252"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187679-t-compiler/wg-parallel-rustc/topic/slowdown%20compiling%20Cargo/near/176509252" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> cuviper <a href="https://rust-lang.github.io/zulip_archive/stream/187679-t-compiler/wg-parallel-rustc/topic/slowdown.20compiling.20Cargo.html#176509252">(Sep 24 2019 at 21:41)</a>:</h4>
<p>rayon creates threads for the global pool on first use</p>



<a name="176509253"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187679-t-compiler/wg-parallel-rustc/topic/slowdown%20compiling%20Cargo/near/176509253" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Alex Crichton <a href="https://rust-lang.github.io/zulip_archive/stream/187679-t-compiler/wg-parallel-rustc/topic/slowdown.20compiling.20Cargo.html#176509253">(Sep 24 2019 at 21:41)</a>:</h4>
<p><a href="/user_uploads/4715/R8TIiPYEW87HhqYF0MGjELss/Capture.PNG" target="_blank" title="Capture.PNG">Capture.PNG</a> </p>
<div class="message_inline_image"><a href="/user_uploads/4715/R8TIiPYEW87HhqYF0MGjELss/Capture.PNG" target="_blank" title="Capture.PNG"><img src="/user_uploads/4715/R8TIiPYEW87HhqYF0MGjELss/Capture.PNG"></a></div><p>this is a small snippet of the perf sorted by self-time</p>



<a name="176509275"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187679-t-compiler/wg-parallel-rustc/topic/slowdown%20compiling%20Cargo/near/176509275" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> cuviper <a href="https://rust-lang.github.io/zulip_archive/stream/187679-t-compiler/wg-parallel-rustc/topic/slowdown.20compiling.20Cargo.html#176509275">(Sep 24 2019 at 21:41)</a>:</h4>
<p>or if you create a manual ThreadPool, it's new threads for each time you do so</p>



<a name="176509277"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187679-t-compiler/wg-parallel-rustc/topic/slowdown%20compiling%20Cargo/near/176509277" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Alex Crichton <a href="https://rust-lang.github.io/zulip_archive/stream/187679-t-compiler/wg-parallel-rustc/topic/slowdown.20compiling.20Cargo.html#176509277">(Sep 24 2019 at 21:41)</a>:</h4>
<p>hm ok, I'll open an issue for that</p>



<a name="176509344"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187679-t-compiler/wg-parallel-rustc/topic/slowdown%20compiling%20Cargo/near/176509344" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/187679-t-compiler/wg-parallel-rustc/topic/slowdown.20compiling.20Cargo.html#176509344">(Sep 24 2019 at 21:42)</a>:</h4>
<p><span class="user-mention" data-user-id="138448">@cuviper</span> so to be clear we would expect num_cpus threads per ThreadPool creation, approximately, right?</p>



<a name="176509360"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187679-t-compiler/wg-parallel-rustc/topic/slowdown%20compiling%20Cargo/near/176509360" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/187679-t-compiler/wg-parallel-rustc/topic/slowdown.20compiling.20Cargo.html#176509360">(Sep 24 2019 at 21:42)</a>:</h4>
<p><span class="user-mention" data-user-id="116015">@Alex Crichton</span> I wonder if this is the jobserver threads -- IIRC, there's a thread spawn in that crate?</p>



<a name="176509370"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187679-t-compiler/wg-parallel-rustc/topic/slowdown%20compiling%20Cargo/near/176509370" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> cuviper <a href="https://rust-lang.github.io/zulip_archive/stream/187679-t-compiler/wg-parallel-rustc/topic/slowdown.20compiling.20Cargo.html#176509370">(Sep 24 2019 at 21:42)</a>:</h4>
<p>the number is tunable, but it defaults to number of cpus, yes</p>



<a name="176509429"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187679-t-compiler/wg-parallel-rustc/topic/slowdown%20compiling%20Cargo/near/176509429" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Alex Crichton <a href="https://rust-lang.github.io/zulip_archive/stream/187679-t-compiler/wg-parallel-rustc/topic/slowdown.20compiling.20Cargo.html#176509429">(Sep 24 2019 at 21:43)</a>:</h4>
<p><span class="user-mention" data-user-id="116122">@simulacrum</span> yes there's one thread in the jobserver crate</p>



<a name="176509439"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187679-t-compiler/wg-parallel-rustc/topic/slowdown%20compiling%20Cargo/near/176509439" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Alex Crichton <a href="https://rust-lang.github.io/zulip_archive/stream/187679-t-compiler/wg-parallel-rustc/topic/slowdown.20compiling.20Cargo.html#176509439">(Sep 24 2019 at 21:43)</a>:</h4>
<p>but I suspect the 28 threads spawned by each rustc is dwarfing that</p>



<a name="176509454"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187679-t-compiler/wg-parallel-rustc/topic/slowdown%20compiling%20Cargo/near/176509454" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Alex Crichton <a href="https://rust-lang.github.io/zulip_archive/stream/187679-t-compiler/wg-parallel-rustc/topic/slowdown.20compiling.20Cargo.html#176509454">(Sep 24 2019 at 21:43)</a>:</h4>
<p>this is an aggregate of all rustc processes near the start of the build</p>



<a name="176509457"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187679-t-compiler/wg-parallel-rustc/topic/slowdown%20compiling%20Cargo/near/176509457" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Alex Crichton <a href="https://rust-lang.github.io/zulip_archive/stream/187679-t-compiler/wg-parallel-rustc/topic/slowdown.20compiling.20Cargo.html#176509457">(Sep 24 2019 at 21:43)</a>:</h4>
<p>(that profile I linked above)</p>



<a name="176509465"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187679-t-compiler/wg-parallel-rustc/topic/slowdown%20compiling%20Cargo/near/176509465" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> cuviper <a href="https://rust-lang.github.io/zulip_archive/stream/187679-t-compiler/wg-parallel-rustc/topic/slowdown.20compiling.20Cargo.html#176509465">(Sep 24 2019 at 21:43)</a>:</h4>
<p>the pool is created here:<br>
<a href="https://github.com/rust-lang/rust/blob/66bf391c3aabfc77f5f7139fc9e6944f995d574e/src/librustc_interface/util.rs#L209-L213" target="_blank" title="https://github.com/rust-lang/rust/blob/66bf391c3aabfc77f5f7139fc9e6944f995d574e/src/librustc_interface/util.rs#L209-L213">https://github.com/rust-lang/rust/blob/66bf391c3aabfc77f5f7139fc9e6944f995d574e/src/librustc_interface/util.rs#L209-L213</a></p>



<a name="176509467"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187679-t-compiler/wg-parallel-rustc/topic/slowdown%20compiling%20Cargo/near/176509467" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Alex Crichton <a href="https://rust-lang.github.io/zulip_archive/stream/187679-t-compiler/wg-parallel-rustc/topic/slowdown.20compiling.20Cargo.html#176509467">(Sep 24 2019 at 21:43)</a>:</h4>
<p>and so if you spawn 28 rustc's that each spawn 28 threads</p>



<a name="176509469"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187679-t-compiler/wg-parallel-rustc/topic/slowdown%20compiling%20Cargo/near/176509469" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Alex Crichton <a href="https://rust-lang.github.io/zulip_archive/stream/187679-t-compiler/wg-parallel-rustc/topic/slowdown.20compiling.20Cargo.html#176509469">(Sep 24 2019 at 21:43)</a>:</h4>
<p>that's a lot of threads to spawn very quickly</p>



<a name="176509473"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187679-t-compiler/wg-parallel-rustc/topic/slowdown%20compiling%20Cargo/near/176509473" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/187679-t-compiler/wg-parallel-rustc/topic/slowdown.20compiling.20Cargo.html#176509473">(Sep 24 2019 at 21:43)</a>:</h4>
<p>ah -- so we're spawning like 800 threads :)</p>



<a name="176509529"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187679-t-compiler/wg-parallel-rustc/topic/slowdown%20compiling%20Cargo/near/176509529" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Alex Crichton <a href="https://rust-lang.github.io/zulip_archive/stream/187679-t-compiler/wg-parallel-rustc/topic/slowdown.20compiling.20Cargo.html#176509529">(Sep 24 2019 at 21:44)</a>:</h4>
<p>the crate at the beginning of the crate graph are also very quick to compile, on the order of a hundred or so ms</p>



<a name="176509540"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187679-t-compiler/wg-parallel-rustc/topic/slowdown%20compiling%20Cargo/near/176509540" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Alex Crichton <a href="https://rust-lang.github.io/zulip_archive/stream/187679-t-compiler/wg-parallel-rustc/topic/slowdown.20compiling.20Cargo.html#176509540">(Sep 24 2019 at 21:44)</a>:</h4>
<p>so we're thrashing thread creation quite a lot</p>



<a name="176509553"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187679-t-compiler/wg-parallel-rustc/topic/slowdown%20compiling%20Cargo/near/176509553" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/187679-t-compiler/wg-parallel-rustc/topic/slowdown.20compiling.20Cargo.html#176509553">(Sep 24 2019 at 21:44)</a>:</h4>
<p>I wonder if we could get rayon to sort of "slow start" thread spawning</p>



<a name="176509578"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187679-t-compiler/wg-parallel-rustc/topic/slowdown%20compiling%20Cargo/near/176509578" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/187679-t-compiler/wg-parallel-rustc/topic/slowdown.20compiling.20Cargo.html#176509578">(Sep 24 2019 at 21:45)</a>:</h4>
<p>e.g. create the pool with size 1 and then if it detects work after 1 second grow to num_cpus</p>



<a name="176509608"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187679-t-compiler/wg-parallel-rustc/topic/slowdown%20compiling%20Cargo/near/176509608" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> cuviper <a href="https://rust-lang.github.io/zulip_archive/stream/187679-t-compiler/wg-parallel-rustc/topic/slowdown.20compiling.20Cargo.html#176509608">(Sep 24 2019 at 21:45)</a>:</h4>
<p>we've talked about dynamic threads before, but that's ... challenging</p>



<a name="176509677"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187679-t-compiler/wg-parallel-rustc/topic/slowdown%20compiling%20Cargo/near/176509677" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> cuviper <a href="https://rust-lang.github.io/zulip_archive/stream/187679-t-compiler/wg-parallel-rustc/topic/slowdown.20compiling.20Cargo.html#176509677">(Sep 24 2019 at 21:46)</a>:</h4>
<p>I guess you're not talking fully dynamic though, just lazy</p>



<a name="176509684"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187679-t-compiler/wg-parallel-rustc/topic/slowdown%20compiling%20Cargo/near/176509684" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/187679-t-compiler/wg-parallel-rustc/topic/slowdown.20compiling.20Cargo.html#176509684">(Sep 24 2019 at 21:46)</a>:</h4>
<p>right, yeah</p>



<a name="176509714"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187679-t-compiler/wg-parallel-rustc/topic/slowdown%20compiling%20Cargo/near/176509714" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Alex Crichton <a href="https://rust-lang.github.io/zulip_archive/stream/187679-t-compiler/wg-parallel-rustc/topic/slowdown.20compiling.20Cargo.html#176509714">(Sep 24 2019 at 21:46)</a>:</h4>
<p><a href="https://github.com/rust-lang/rust/issues/64752" target="_blank" title="https://github.com/rust-lang/rust/issues/64752">https://github.com/rust-lang/rust/issues/64752</a></p>



<a name="176562781"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187679-t-compiler/wg-parallel-rustc/topic/slowdown%20compiling%20Cargo/near/176562781" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Zoxc <a href="https://rust-lang.github.io/zulip_archive/stream/187679-t-compiler/wg-parallel-rustc/topic/slowdown.20compiling.20Cargo.html#176562781">(Sep 25 2019 at 13:14)</a>:</h4>
<p>28 threads is also probably not the ideal number to use for rustc due to contention, etc.</p>



<a name="176587147"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187679-t-compiler/wg-parallel-rustc/topic/slowdown%20compiling%20Cargo/near/176587147" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> cuviper <a href="https://rust-lang.github.io/zulip_archive/stream/187679-t-compiler/wg-parallel-rustc/topic/slowdown.20compiling.20Cargo.html#176587147">(Sep 25 2019 at 17:26)</a>:</h4>
<p>maybe rustc needs a heuristic max on its threads?</p>



<a name="176587162"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187679-t-compiler/wg-parallel-rustc/topic/slowdown%20compiling%20Cargo/near/176587162" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> cuviper <a href="https://rust-lang.github.io/zulip_archive/stream/187679-t-compiler/wg-parallel-rustc/topic/slowdown.20compiling.20Cargo.html#176587162">(Sep 25 2019 at 17:26)</a>:</h4>
<p>just like codegen-units uses 16 regardless</p>



<a name="176622860"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187679-t-compiler/wg-parallel-rustc/topic/slowdown%20compiling%20Cargo/near/176622860" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nagisa <a href="https://rust-lang.github.io/zulip_archive/stream/187679-t-compiler/wg-parallel-rustc/topic/slowdown.20compiling.20Cargo.html#176622860">(Sep 26 2019 at 02:16)</a>:</h4>
<p>We also should have a minimum on job tokens that a rustc holds</p>



<a name="176622866"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187679-t-compiler/wg-parallel-rustc/topic/slowdown%20compiling%20Cargo/near/176622866" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nagisa <a href="https://rust-lang.github.io/zulip_archive/stream/187679-t-compiler/wg-parallel-rustc/topic/slowdown.20compiling.20Cargo.html#176622866">(Sep 26 2019 at 02:16)</a>:</h4>
<p>like a rustc should hold at least 1 token always.</p>



<a name="176622874"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187679-t-compiler/wg-parallel-rustc/topic/slowdown%20compiling%20Cargo/near/176622874" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nagisa <a href="https://rust-lang.github.io/zulip_archive/stream/187679-t-compiler/wg-parallel-rustc/topic/slowdown.20compiling.20Cargo.html#176622874">(Sep 26 2019 at 02:17)</a>:</h4>
<p>avoids spawning 56 rustcs on a 8 core system.</p>



<a name="176679656"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187679-t-compiler/wg-parallel-rustc/topic/slowdown%20compiling%20Cargo/near/176679656" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nikomatsakis <a href="https://rust-lang.github.io/zulip_archive/stream/187679-t-compiler/wg-parallel-rustc/topic/slowdown.20compiling.20Cargo.html#176679656">(Sep 26 2019 at 17:46)</a>:</h4>
<p>btw I remain fairly unconvinced that rayon is a good fit for us, in general</p>



<a name="176679662"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187679-t-compiler/wg-parallel-rustc/topic/slowdown%20compiling%20Cargo/near/176679662" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nikomatsakis <a href="https://rust-lang.github.io/zulip_archive/stream/187679-t-compiler/wg-parallel-rustc/topic/slowdown.20compiling.20Cargo.html#176679662">(Sep 26 2019 at 17:46)</a>:</h4>
<p>much as I love rayon of course :)</p>



<a name="176679677"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187679-t-compiler/wg-parallel-rustc/topic/slowdown%20compiling%20Cargo/near/176679677" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nikomatsakis <a href="https://rust-lang.github.io/zulip_archive/stream/187679-t-compiler/wg-parallel-rustc/topic/slowdown.20compiling.20Cargo.html#176679677">(Sep 26 2019 at 17:46)</a>:</h4>
<p>I think it might eventually be a good fit, but I sort of suspect we might also do better with some simpler setup for the time being</p>



<a name="176686920"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187679-t-compiler/wg-parallel-rustc/topic/slowdown%20compiling%20Cargo/near/176686920" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/187679-t-compiler/wg-parallel-rustc/topic/slowdown.20compiling.20Cargo.html#176686920">(Sep 26 2019 at 19:02)</a>:</h4>
<p>I think we might get away with managing the thread pool ourselves but otherwise using rayon, though I don't know how possible/realistic that is</p>



<hr><p>Last updated: Aug 07 2021 at 22:04 UTC</p>
</html>